Computer-generated imagery typically involves using software and/or hardware to generate one or more images from a geometric model. A geometric model defines objects, light sources, and other elements of a virtual scene and a rendering system or other computer/software/hardware system will read in the geometric model and determine what colors are needed in what portions of the image. A renderer will generate a two-dimensional (“2D”) or three-dimensional (“3D”) array of pixel color values that collectively result in the desired image or images.
For a simple geometric model, such as a cube in a vacuum with a single light source, a simple computer program running on most computer hardware could render the corresponding image in a reasonable amount of time without much optimization effort. However, there are many needs—in the entertainment industry and beyond—for methods and apparatus that can efficiently process complex interactions of virtual objects to generate imagery in constrained timeframes where the images might need to convey realistic light interactions, such as light interacting with a participating medium.
Examples of such interaction include the scattering of light, which results in visual complexity, such as when the light interacts with participating media such as clouds, fog, and even air. Rendering this complex light transport typically involves solving a radiative transport equation (see, e.g., [Chandrasekar]) combined with a rendering equation (see, e.g., [Kajiya]) as a boundary condition. Other light interactions might include light passing through objects, such as refractive objects.
Rendering preferably involves computing unbiased, noise-free images. Unfortunately, the typical options to achieve this are variants of brute force path tracing (see, e.g., [Kajiya], [Lafortune93], [Veach], [Lafortune96]) and Metropolis light transport (see, e.g., [Veach], [Pauly]), are notoriously slow to converge to noise-free images despite recent advances (see, e.g., [Raab; Yue]). This becomes particularly problematic when the scene contains specular-diffuse-specular (“SDS”) subpaths or specular-media-specular (“SMS”) subpaths, which are common in physical scenes. These SDS or SMS subpaths may result from diffused reflections and/or refractions of light from a direct light source, giving a scene a realistic, ambient illumination. For example, in a room with a light bulb hanging from the ceiling being the direct light source, light from the light bulb is also reflected off the walls of the room and object surfaces in the room, or light may travel through mediums, such as a glass of water, thus creating SDS and SMS subpaths. The SDS/SMS properties might be used for scenes where there is illumination due to a light source inside a glass fixture or complex light paths, as might be the case with caustics that create light patterns on the bottom of a swimming pool as a result of light shining through the water. Unfortunately, path tracing methods cannot robustly handle these situations, especially in the presence of small light sources.
Methods based on volumetric photon mapping do not suffer from these problems (see, e.g., [Jensen]). They can robustly handle SDS and SMS subpaths, and generally produce less noise. However, these methods suffer from bias. Bias can be eliminated, in theory, by using infinitely many photons, but in practice this is not feasible since tracking infinitely many photons requires unlimited memory.
Volumetric photon mapping is an approach to dealing with light in a participating medium, which is described in [Jensen]. [Jarosz08] describes an improvement to avoid costly and redundant density queries due to ray marching. In those approaches, a renderer formulates a “beam radiance estimate” that considers all photons along the length of a ray in one query. For example, [Jarosz11] showed how to apply the beam concept not just to the query operation but also to the photon data representation. In that approach, the entire photon path is used instead of just photon points, to obtain significant quality and performance improvement. This is similar in spirit to the concept of ray maps for surface illumination (see, e.g., [Lastra], [Havran], [Herzog]), as well as the recent line-space gathering technique (see, e.g., [Sun]). Those methods result in bias, which allows for more efficient simulation; however, when the majority of the illumination is due to caustics (which is often the case with realistic lighting fixtures or when there are specular surfaces), the photons are visualized directly and a large number is required to obtain high-quality results. Though these methods converge to an exact solution as the number of photons increases, obtaining a converged solution requires storing an infinite collection of photons, which is not feasible.
Progressive photon mapping, as shown in, for example, [Hachisuka08], alleviates this memory constraint. Instead of storing all photons needed to obtain a converged result, it updates progressive estimates of radiance at measurement points in the scene (see, e.g., [Hachisuka08], [Hachisuka09]) or on the image plane (see, e.g., [Knaus]). Photons are traced and discarded progressively, and the radiance estimates are updated after each photon tracing pass in such a way that the approximation converges to the correct solution in the limit.
Previous progressive techniques have primarily focused on surface illumination, but [Knaus] demonstrated results for the traditional volumetric photon mapping as in [Jensen]. Unfortunately, volumetric photon mapping with photon points produces inferior results compared to photon beams as explained by [Jarosz11], especially in the presence of complex specular interactions that benefit most from progressive estimation.
Light interactions in participating media are responsible for many subtle but rich lighting effects in fire, water, smoke, clouds and crepuscular rays. Unfortunately, accurately rendering these effects is very costly. [Cerezo] provides a comprehensive overview of recent techniques.
Though methods such as (bidirectional) path tracing (see, e.g., [Lafortune93], [Veach], [Lafortune96]) or Metropolis light transport (see, e.g., [Pauly]) are general and provide unbiased solutions, it is costly to obtain high-quality, noise-free solutions except in the simplest scenes. The most successful approaches typically combine photon tracing with a biased Monte Carlo framework (see, e.g., [Jensen], [Walter06], [Jarosz08]).
Volumetric photon mapping traces paths from light sources and stores “photons” at the vertices of these paths (see, e.g., [Jensen]). A subsequent rendering pass approximates lighting using density estimation of the photons. The recent beam radiance estimate (BRE), as shown in [Jarosz08], for photon mapping further accelerates this approach and improves the rendering quality by accounting for all photons that are located along a ray query.
However, even with such state-of-the-art optimizations, photon mapping suffers from a few remaining sources of inefficiency: 1) complex high-frequency lighting requires an excessive number of photons, which increases memory demands and leads to slow rendering times; 2) many photons may be processed which ultimately contribute very little to the final pixel color (due to subsequent attenuation or small projected pixel size); and 3) since photons are generated stochastically, the reconstructed density is prone to flickering unless many photons are used.
Thus, even with those techniques, improvements to existing rendering techniques are needed.